ZEPPELIN-497 ] pyspark completion by cloverhearts · Pull Request #530 · apache/zeppelin

cloverhearts · 2015-12-11T10:57:35Z

Added auto-complete for pyspark.
autocomplete function will operate through the spark python tag '% pyspark'.
short-cut = Ctrl + Shift + Space.

jira issue : https://issues.apache.org/jira/browse/ZEPPELIN-497

felixcheung · 2015-12-13T05:36:58Z

spark/src/main/resources/python/zeppelin_pyspark.py

if text_value is None should it returned all objects/names in the current scope (dir())?

elbamos · 2015-12-13T06:18:24Z

@felixcheung

Miscellaneous requests for improvement belong in jiras.

Rather than giving everyone suggestions on what to do in their PRs, perhaps you should try making them as PRs of your own.

felixcheung · 2015-12-13T20:29:30Z

tested and working well! thanks for working on this

prabhjyotsingh · 2015-12-16T06:47:18Z

Tested, works well 👍

cloverhearts · 2015-12-16T16:42:35Z

Filtering for FunctionName added.
Thank you for Feed back.

Leemoonsoo · 2015-12-16T22:59:25Z

Working well. Thanks for very useful improvement.
LGTM, Merge if there're no more discussions.

felixcheung · 2015-12-16T23:20:48Z

@cloverhearts thanks - to clarify the rule is two leading underscores (__foo)
https://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables
"To avoid name clashes with subclasses, use two leading underscores to invoke Python's name mangling rules. Python mangles these names with the class name: if class Foo has an attribute named __a , it cannot be accessed by Foo.__a"

you do not match both leading and trailing (__foo__)

cloverhearts · 2015-12-17T01:54:23Z

Changed Filter Rule.

Filter __FunctionName

NAME The form is not a target filter.

felixcheung · 2015-12-17T03:53:41Z

sorry if it's confusing - the method name mangling rule in Python applies to any name that starts with __, so it includes both __foo and __foo__ (eg. __getattr__)

so I think the regex should be ^__.+ or ^__.*

cloverhearts · 2015-12-17T04:06:08Z

@felixcheung
Sorry, I just modified. Tested after upload.

cloverhearts · 2015-12-17T04:56:49Z

Filtering

NAME and __NAME

Tested.

felixcheung · 2015-12-17T05:27:41Z

looks good - thank you!

cloverhearts · 2015-12-17T05:28:36Z

@felixcheung Thank you for Feed back! have a nice day!

Leemoonsoo · 2015-12-18T04:12:22Z

LGTM
Merge if there're no more discussions

fanlu · 2016-09-26T14:23:12Z

version 0.7.0-SNAPSHOT
build by mvn clean package -Pbuild-distr -Pspark-2.0 -Dspark.version=2.0.0 -Pscala-2.11 -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.7.0 -Pyarn -Psparkr -Ppyspark -DskipTests -Drat.skip=true -U
when I use ctrl+. the stacktrace is
ERROR [2016-09-26 22:02:25,834]({qtp532445947-596} NotebookServer.java[onMessage]:257) - Can't handle message
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:453)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.completion(LazyOpenInterpreter.java:122)
at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:242)
at org.apache.zeppelin.notebook.Note.completion(Note.java:520)
at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:793)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:216)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_completion(RemoteInterpreterService.java:358)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.completion(RemoteInterpreterService.java:342)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:449)
... 21 more

felixcheung · 2016-09-26T17:41:48Z

hi @fanlu - it's best to report issue to dev@zeppelin.apache.org

astroshim · 2016-09-27T16:11:54Z

@fanlu As @felixcheung mentioned, you can report issue to dev@zeppelin.apache.org.
and Could you test #1465 for solving your problem?

fanlu · 2016-09-28T15:26:42Z

@astroshim
sorry, the problem is still there
#1465 (comment)

### What is this PR for? fanlu reported bug on #530. This PR fixes bug of checking condition in PySparkInterpreter. ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1500 ### How should this be tested? Outline the steps to test the PR here. ### Questions: - Does the licenses files need update? no - Is there breaking changes for older versions? no - Does this needs documentation? no Author: astroshim <hsshim@nflabs.com> Closes #1465 from astroshim/ZEPPELIN-1500 and squashes the following commits: d442c12 [astroshim] fix to wait completion data e50606a [astroshim] Merge branch 'master' into ZEPPELIN-1500 cb4b0fb [astroshim] add testcase 7e93968 [astroshim] Merge branch 'ZEPPELIN-1500' of https://github.com/astroshim/zeppelin into ZEPPELIN-1500 49282b7 [astroshim] remove NullPointException 60da1c3 [astroshim] fix condition ee2a29b [astroshim] fix bug of condition check.

implement pyspark completion

ab05f86

felixcheung reviewed Dec 13, 2015
View reviewed changes

cloverhearts added 2 commits December 16, 2015 10:08

Pyspark completion __ filtering

f4dc487

remove variable resultComplaetion

1f82825

change filter regex character

d7efe88

Changed PySpark Completion Filter regex

6832409

asfgit closed this in 81b1e22 Dec 20, 2015

astroshim mentioned this pull request Sep 27, 2016

[ZEPPELIN-1500] pyspark completion doesn't work. #1465

Closed

Conversation

cloverhearts commented Dec 11, 2015

Uh oh!

felixcheung Dec 13, 2015

Choose a reason for hiding this comment

Uh oh!

elbamos commented Dec 13, 2015

Uh oh!

felixcheung commented Dec 13, 2015

Uh oh!

prabhjyotsingh commented Dec 16, 2015

Uh oh!

cloverhearts commented Dec 16, 2015

Uh oh!

Leemoonsoo commented Dec 16, 2015

Uh oh!

felixcheung commented Dec 16, 2015

Uh oh!

cloverhearts commented Dec 17, 2015

Changed Filter Rule.

Uh oh!

felixcheung commented Dec 17, 2015

Uh oh!

cloverhearts commented Dec 17, 2015

Uh oh!

cloverhearts commented Dec 17, 2015

Filtering

Uh oh!

felixcheung commented Dec 17, 2015

Uh oh!

cloverhearts commented Dec 17, 2015

Uh oh!

Leemoonsoo commented Dec 18, 2015

Uh oh!

fanlu commented Sep 26, 2016

Uh oh!

felixcheung commented Sep 26, 2016

Uh oh!

astroshim commented Sep 27, 2016

Uh oh!

fanlu commented Sep 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants